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Abbreviations 
P300 

ERP 

EEG 
Cont- Bench 
Disc-Bench 
DiscTrim-Bench 



Rand- Bench 



BCI 



BBCI 



It refers to the positive peak created in response to a particu- 
lar stimulus, a usual way to elicit this wave is via the oddball 
paradigm 

Short for Event Related Potential. Event related potentials are 
peaks in waves in epoch data by the use of electroencephalogra- 
phy. 

Common abbreviation for electroencephalography, neuroimaging 
technique used in this study. 

Comparison bewteen two trajectories corresponding to the con- 
tinuous condition and the benchmark - Joystick condition. 

Comparison bewteen two trajectories corresponding to the dis- 
crete condition and the benchmark - Joystick condition. 

Comparison bcwtccn two trajectories corresponding to the Trimmed 
Discrete Condition and the benchmark - Joystick condition. The 
trimming has to do with the data processing that deleted the 
points in the discrete trajectory where the participant was not 
moving. See Section 4.1.2. 

Comparison bewteen two trajectories corresponding one of the 
120 random trajectories generated with recorded data and the 
benchmark - Joystick condition. 

Short form commonly used to describe a brain computer inter- 
face. 

Berlin Brain Computer Interface group. This abbreviation is 
most commonly used in this paper when it refers to the Berlin 
Brain Computer Interface toolbox whose MATLAB functions 



ix 



and routines were used in the experimental design during: sig- 
nal acquisition, calibration, classification as well as the recorded 
EEG signals from the people who participated in this study. 

UDP The User Datagram Protocol (UDP) is one of the core members 
of the Internet protocol suite. With UDP, computer applica- 
tions can send messages, in this case referred to as datagrams, 
to other hosts on an Internet Protocol (IP) network without 
prior communications to set up special transmission channels or 
data paths, according to Wikipedia. They are useful for this 
experiment because UDPs are the output of choice of the BBCI 
toolbox and are used to send signals to feedback applications. 

OSC Open Sound Control is a content format for messaging among 
computers, sound synthesizers, and other multimedia devices 
that are optimized for modern networking technology, accord- 
ing to Wikipedia. In this particular experiment OSC signals 
were used because they were required to serve as input to the 
virtual joystick that controlled the feedback application. 
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Introduction 



The technology created by Brain Computer Interfaces (BCI) for navigational pur- 
poses, belongs to the area of neuroprosthetics, technology developed as tools for 
patients with motor disabilities. Certainly, novel applications that have seen the 
light such as BCI controlled Robots and manipulation of objects in three dimensions 
arc not necessarily developed for clinical populations specifically. These applications 
include models of drones or helicopters controlled with BCIs that fly in 3D envi- 
ronments (14; 2) which simulate a real flight. Recently BCI controlled flight was 
achieved with a quadcopter developed at the University of Minnesota, whose tech- 
nology, according to the news release: "goes far beyond fun and games and has the 
potential to help people who are paralyzed or have neurodegenerative diseases" ^. 
Other innovative use of BCIs is the development of games that have people move 
objects with the help of BCI inspired technology^. 

Commonly, navigational BCIs use motor imagery and visually evoked ERPs (1; 
2; 10) to create signals that allow robots or navigational programs to move. For 

^ . . . http : //wwwl . umn . edu/iiews/news-releases/2013/UR_C0NTENT_445216 . html 

^ . . . http : //www . kickstarter . com/pro j ects/ 1544851629/ throw-trucks-with-your-mind 
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several reasons such as physical limitations in participants, additional modalities 
other than motor imagery have been and must be explored. One of said models is 
the AMUSE paradigm which makes use of binary classification that also comes from 
added spatial information or "spatial hearing" that makes it possible to use the same 
spatial information to guide object motion in a three dimensional environment. 

The occurrence of the wave P300 is the electrical manifestation product of a 
complex set of ERPs evoked artificially by the oddball paradigm. In the oddball 
paradigm, a target previously recognized, in our case, a sound, will be presented 
among other sounds meant as non-targets in a manner similar to the well known 
cocktail party effect. Due to the short latency of the P300 wave, a fast presentation 
of stimuli to the person is possible by stocking stimulus after stimulus with short 
inter stimulus intervals, and still being possible to classify said waves for further 
use as it has been shown by Schreuder and others in previous studies (2010, 2011). 
Other examples of BCI applications include: BCI controlled virtual 3D helicopters 
using intelligent control strategies (14), Asynchronous BCI controlled car in a virtual 
reality environment (18), Flying simulated aircraft with EEG (1) and sketches of a 
P300 BCI based movement (8) mostly designed for neuro-prosthetics and wheelchair 
applications, in which auditory BCIs are discussed and the potentials advantages 
and disadvantages are discussed. 

The present theoretical framework, however, does correspond concretely to the 
theories developed by Schreuder, Hohne, Tangermann and Blankertz (17; 16) at TU 
Berlin. Their ideas explore the development of Brain Computer Interface applications 
focused mainly in auditory stimulation and BCI spellers, technology which, can be 
further probed and be helpful for people with certain handicaps, for example, for 
the use with patients suffering from amyotrophic lateral sclerosis (17) for improving 
their possibilities of communication (with the use of the AMUSE speller for example. 
The AMUSE paradigm makes use of multiple binary classifications using auditory 
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inputs so that participants can have control over word speUing). 
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Theoretical Framework 



2.1 Oddball Paradigm and P300 Elicitation 

P300 is a wave present in the human brain which gets ehcited in response to a rare 
or unpredicted event in almost any case without prior training. In order to ehcit a 
P300 wave, the oddball paradigm is the most common method used. The oddball 
paradigm has been used in the previous experiments by Schreuder, Tangermann and 
Blankertz (17) and it has been reported to have positive P300 wave generation in 
the brains of participant. Similarly, there have been studies where P300 was elicited 
by visual stimulation, tactile stimulation, auditory and a combination of both, visual 
stimulation and auditory stimulation (10). In the auditory case, the method is the 
following: letters flashing on a screen were used so that participants evoke P300 
responses that can be used to train classifiers and posteriorly use the classifiers to 
compare novel stimuli online. Another experiment was created in which, rows and 
columns of letters were presented, as in the previous experiment but the letters did 
not flash, they instead remained unchanged but shown just for reference, in turn 
participants were told to pay attention to a voice that cued the desired target (10). 
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Figure 2.1: ERP Complex, ERP components, PlOO, N200 and 
P300 are marked with red for better visualization. Taken from url 
http://scan.oxfordjournals.Org/content/2/4/323/F2.expansion querry: Oct 27 
2013 

Both are examples of proven methods to create P300s. In the current study, auditory 
stimulation using the best experimental conditions and design were chosen to create 
a viable experimental setup that can be used for P300 spatial navigation, namely 
the use of spatial hearing and the AMUSE paradigm. 

2.2 Spatial Hearing 

According to Schreuder (2010) the direction that the acoustic stimuli come from 
adds additional information about the stimulus that elicits the P300s waves in the 
brain. The advantage of the Spatial Hearing is that: "Localization of sounds in 
space is [a] process that our brain does without mental effort, several studies show 
the ability of human listeners to distinguish sounds in space when subjects focus 
on a particular direction, [attention appears] to be distributed in a gradient, with 
decreasing alertness, when [they are] moving away from the attended direction" 
(17). With this idea in mind is it possible to create a navigational ERP based BCI 
on spatial hearing, in which, the directions used for spatial hearing will be the same 
used for navigation later on. In order to achieve an experiment based on spatial 
hearing the desired locations will have to be placed accordingly. 



5 



source 



signal 



feature 



classifier 



control 



feedback 




acquire ccrtiniioLis segmentation apply 

biosignals processing 'eaure classsfier 

extraction 



Spacial 
Control 

Strategy 



Virtual joystick 
rnovement geriErated 
by BCI on IVIazeWall<er 



classifier 
output 



control feedback | 
signal applicatioj/ 



Figure 2.2: Online BCI data pipeline. Adapted from the Wiki of the IDA Group 



According to Hohne 2011 (12) "the AMUSE paradigm (Auditory MUlticlass Spatial 
ERP), as presented in [Schreuder et al., 2010], was adapted for online text writing. 
It consists of a ring with 6 audio speakers around the subject, which produces stimuli 
in a pseudo random order. The inter stimulus onset interval (ISOI) was 175ms with 
each tone lasting 40ms" . 

The AMUSE paradigm has six steps: sourcing, signal processing, feature ex- 
traction, classification, control and feedback. For this study, only four of the steps 
were taken from the AMUSE paradigm described by Hohne et. al (2011). Con- 
trol strategies and feedback phase due to the navigational nature of the study were 
transformed. The mentioned steps had to be altered in the following way: a) in the 
control phase a new control strategy was created that permitted participants make 
use of their signals to move around in the maze and b) the signals were sent to the 
feedback application which consisted of a 3D game engine that was developed for 
maze design, analysis and presentation 

2.4 ERP Spatial Hearing Based Navigation 

Spatial hearing based navigation can be explained in the following way: by taking 
the classification scores generated by an algorithm or classifier previously chosen. 



2.3 AMUSE Paradigm 
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Depending on the accuracy of the algorithm and the classification score particular to 

the participant, it is possible to generate meaningful movement to a certain direction 
selected by the participant. It is possible to make use of two different modalities to 
achieve movement: 

2.4-1 Continuous Online ERP Classification 

Taking the ERP classification scores from a participant and appending them con- 
tinuously in a unified stream of classification data, a movement command can be 
produced in the following way. Based on the classification data fiow, the challenge 
is to create an algorithm that deals with dimensionality dissimilarities and sifts in- 
coming data into a stream of movement commands in two or even more dimensions 
if desired by simply comparing the stream of data to the calibration data recorded 
previously. The translation from classification values into movement or vector values 
will have to take into consideration not only the most recent classification but older 
classifications as well. So having a window of classifications from the most recent 
toward the past is the way in which the control algorithm makes sense of what di- 
rection a participant wants to go to. In bio-signal terms, this means, to select EEG 
calibration bio-signals that resemble more to the window of more recent online sig- 
nals. By using Continuous Online ERP Classification, it will depend on the number 
of speakers assigned for a certain dimension whether the BCI will is able to move in 
a 2D environment or a 3D environment. 

2.4.2 Discrete Online ERP Classification 

Taking the ERP classification scores from a participant in a single round of classifica- 
tion and by grouping them in a single block of online classification data, a movement 
command can be deduced. Based on a block of data from a classification round of 
n-iterations, we will run an algorithm that simply deduces the direction intended 
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to move giving commands only in the direction the participant desired to move in a 
binary fashion (either go to a certain direction or not). In the same way with Contin- 
uous Onhne ERP Classification, it will depend on the number of speakers assigned for 
a certain dimension whether the BCI will is able to move in a 2D environment or a 3D 
environment; only that under this condition it will be able to move in a stcp-by-stcp 
fashion, or more precisely by calculating classification-round-by-classification-round. 
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3 

Experimental Methods 



3.1 Experimental setup and Data Acquisition 

The present study was conducted under supervision of Michael Tangermann and 
Martijn Schreuder. All experiments were conducted within the laboratories of the 
BBCI group of the Machine Learning department at the Berlin Institute of Technol- 
ogy. The participants provided their written informed consent to participate in the 
study according to the approved procedures set forth by the lab and approved by 
this study's supervisor. 

Seven participants, five females and two males between the ages of twenty one 
and forty with a mean of 29 years old took part in the experiment. Only one of them 
had previous experience with auditory BCIs. The feedback was generated by Maze- 
Walker with custom build mazes. The appearance and demeanor of Mazewalker was 
similar to that of a video game. For that reason a screen was positioned in front 
of the paticipants. To facilitate movement and spatial hearing this setup can be 
described as: Participants sat in front of a computer screen. They were surrounded 
by five active speakers pointing to 90, 45, 0, 135 and 180 degrees from the partici- 
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Figure 3.1: Schematic of the experimental setup. Adapted from Schreuder et.al. 
2010. Each speaker represents a desired direction in the maze shown in Figure 2 




Figure 3.2: A screen capture taken in the first level of the house maze. Movement 
controls are: X Axis: Rotating Left: speaker 4; Rotating right: speaker 3. Z Axis: 
Moving forward: speaker 1;. Combined Movements: going forward while turning 
left: speaker 5; going forward while turning right: speaker 2 

pants perspective (Fig. 3.1). The distance separating the participants head and the 
speakers was around 65 cm, this is to assure that the spatial hearing locations will 
resemble the most to the original experiment and follow ups by Schreuder and others. 
The monitor will show a 1st person perspective in a 3D environment through which 
participants were asked to navigate, the images of the 3-Dimensional environment 
will similar to the one shown in Fig 3.2 

Speaker number four and number three corresponded to the X axis. Whenever 
the participant paid attention to either of these speakers the camera will rotate to 
either the left (represented by the speaker four) or to the right (represented by the 
speaker three). In Turn whenever a participant wanted to move forward, he was 
asked to attend to the frontal speaker. Left-forward and right-forward movements 
were also possible by dividing the classification score into X and Y values. In the 
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online phase of the experiment, onhne stream of bio signals that source from the EEG 
will be classified and after passing it through a spatial control strategy algorithm, 
the BCI will be able to recognize the desired location of the movement which will be 
the same as the speaker the participant is paying attention to. 

EEG was recorded using a fixed set of 56 Ag/AgCl electrodes and BrainAmp 
amplifiers (Brain Products, Munich, Germany). Electrooculogram (EOG) was co- 
recorded with two bipolar channels. All impedances were kept below 15 kohms.) 
with 32 wet electrodes placed at symmetrical positions based on the International 
10 - 20 system. Data acquisition software (Brain Vision) was used. The signals were 
sampled at 1 kHz and filtered by a hardware analog band-pass filter between 0.1 
and 250 Hz before being digitized and stored for offline analyses. For online use, the 
signal was low-pass filtered below 40 Hz, down sampled to 100 Hz and streamed to the 
online Berlin BCI system. The overall set-up of the BCI and starts with EEG signals 
obtained from the operator via single-montage surface electrodes are processed by 
a computer using windows 7 x64. The nasion was used as reference point. The 
referencing used for the calibration was the same for the online feedback as for the 
offline ERP analysis. Event- related potential classification was performed over these 
signals using the BBCI toolbox. The amuse paradigm initial steps processed the 
data used for classification that produced values that could be used as movement 
inputs afterwards. 

Signal Processing, classification and EEG posterior analysis was only possible 

thanks to the Berlin Brain Computer Interface Toolbox developed by the Machine 

Learning and Intelligent Data Analysis group at the Berlin Institute of Technology. 

For detailed implementations refer to the BBCI - IDA publications on the link below 

^ . The experiment consisted of six blocks which will be described next. 

^ . . .The full list with the publications by the IDA group at TU Berlin is at: http://doc.ml. 
tu-berlin . de/publications/ 
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Figure 3.3: Flowchart explaining the Stages in the Experiment 



3.2 Experimental Task Design 

3.2.1 First Block 

It can also be labeled as free exploration. Participants were asked to explore the 
mazes using a joystick and they also were instructed to find the objects to be col- 
lected, which, in all the mazes, consisted of yellow spheres which turn blue once 
touched. The spheres were at certain places signaling subjects the "optimal" or 
shortest path in the maze, the yellow spheres were included in all the experimental 
as well as in exploration mazes. Additionally, they were asked to learn and remem- 
ber the locations of the rooms, objects and corners of the rooms and the locations 
of where they began and where they ended. The objective of this block was to have 
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the participants try out the maze and help them remember the maze so that they 

know what is present in every maze before any recorded trial started. Data from this 
section was not included in the analyses. . 

3.2.2 Second Block 

Two complete runs of each level using a joystick. The whole experiment had three 
mazes with outdoor and indoor scenarios (See appendix 1). Every level was repeated 
twice by each participant. This measurement was used as the benchmark measure- 
ment, useful to compare the optimal performance that a person can achieve in a maze 
situation using more conventional devices, in this case a joystick. This performance 
was used as the benchmark, as was pointed out earlier on, in order to have a value, 
in this case a trajectory that can be used to compare performance to. We used the 
same mazes presented during the first block. 

3.2.3 Third Block 

Three calibration rounds using auditory stimuli presented and recorded from sub- 
jects. Each speaker had a unique sound identity (easily differentiable from one an- 
other varying in tone and other characteristics) so that directions and location in 
space could be distinguished more easily, so was source discrimination from one an- 
other. The cahbration of signals were done with the exact same specifications of the 
experiments by Schreuder et. al. (2010) and Hohne et. al. (2011). 

During the calibration phase, each trial consisted of 15 pseudo-random auditory 
sequences from each speaker. There was no visual stimulation that could aid clas- 
sification. The presentation of one tone stimulus and the corresponding EEG data 
(with an epoch of 800 ms after auditory stimulation) will be called sub-trial from 
now on. The volume of the speakers was slightly attenuated to help the participants 
concentrate during long periods of auditory BCI navigation because blocks can last 
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up to 10 minutes each. Thus a single cahbration round yielded 5 X 15 sub-trial 
epochs (15 target sub-trials and 4 X 15 non-target subtrials) available for classifier 
training. Participants were asked to count the targets and to report the number of 
occurrences at the end of each trial (counting task). A simple minimum/maximum- 
threshold method was applied to exclude epochs that included artifacts from the 
calibration data: epochs were rejected if their peak-to-peak voltage difference in any 
channel exceeded 100 uV. Data cahbration is done automatically by the toolbox and 
it is not recommended to tweak parameters in the calibration data manually, how- 
ever, there exist other settings belonging to the calibration sub functions but for this 
study the default parameters were kept. Three calibration rounds were performed in 
order for the classifier to ensure performance. 

3.2.4 Fourth Block - Online Sandbox Navigation 

This block consisted of two square-shaped mazes with pictures onto the walls. The 
task, too, was to explore the maze only that by using the BCI. The emphasis was 
on exploring and getting used to the auditory BCI in both conditions: discrete and 
continuous. In order to achieve that, there was one maze with different items for the 
discrete condition and one maze different items for the continuous one. 

3.2.5 Fifth Block - Discrete Auditory BCI 

All three mazes used for the first and second blocks were used for the discrete condi- 
tion as well. It is worth noting though, that the first experimental condition, either 
discrete or continuous was picked at random and that the second maze would be in 
the other experimental condition to alternate conditions. For example: picked at 
random continuous level 1, later discrete level 1, later on continuous level 2, etc. So 
in principle any condition could be picked first and both conditions were alternated 
since then. 
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3.2.6 Sixth Block - Continuous Auditory BCI 

All three mazes used for the first and second blocks were used for the continuous 
condition as well. They were swapped with the discrete ones. 

3.3 Auditory Stimulation 

Stimuh selection for this ERP BCI system was built as follows: five tones varying 
in pitch and tonal character were carefully chosen such that they appear to be as 
different as possible from each other. Tones were presented via five speakers located 
at five different directions already described. Each stimulus lasted 40 ms, The stim- 
ulus onset asynchrony (SOA) was set to 175 ms. The stimulus presentation, the 
online Berlin BCI system and the offline analyses were implemented in MATLAB 
(MathWorks) , making use of the Psychophysics Toolbox^. Off-the-shelf computer 
speakers (type Sony SRS-A201) were the speaker used to produce the stimulation. 
The pseudo-random sequences of stimuli were generated such that two contiguous 
stimuli did not have the same pitch. Moreover, the same stimulus was repeated only 
after at least two other stimuli had appeared. The procedure was meant to resemble 
technical specifications by Schreuder ct. al (2010, 2011). 

3.4 Signal Classification - Calibration 

The cahbration phase consisted of an auditory oddball task where the participants 
were asked to focus on target stimuli while ignoring all non-target stimuli. Partici- 
pants were asked to minimize eye movements and the generation of any other muscle 
artifacts during the whole experiment. It is present during the third block of the 
experiment, the BCI's purpose is to read, separate in epochs and create classifiers 
based on those data and the labels presented. The labels will correspond to the 
^ . . . http : //psychtoolbox . org/HomePage 
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speaker the participants were asked to attend their attention toward. 

Binary classification of target and non-target epochs was performed using a Lin- 
ear discriminant analysis (LDA) . These values represented the mean potential in 
intervals of epochs that were discriminative for the classification task. Due to the 
large dimensionality of features, a shrinkage method (Blankertz et al., 2011) was ap- 
plied to regularize the LDA classifier. Sub-trial epochs, after artifact rejection, were 
used to train the classifier. The classification error for unseen data was estimated by 
cross validation. 

The classifier trigger the classification of the online stream of data. The BCI 
classifier does the procedure iteratively meaning that it will produce five scores, one 
for each speaker. Taking this into account, we will want to have data stored in the 
buffer available for later use. In our case, five classification scores back in time, one 
for each speaker will be enough, other values were tried but the reason for picking this 
particular number was because it provided a balanced tradeoff between accuracy and 
reaction time, in other words, if the data is calculated with more than 5 iterations 
in the past it then will have a higher accuracy but only for steady data and past 
values. So if participants wanted sudden changes of direction which is normal for 
navigation studies the BCI will react too slowly. On the other hand, if the data stored 
is less than three iterations in the past, the BCI will be too inaccurate because there 
wouldnt be enough data to make a meaningful classification or prediction. 

3.5 Feedback 

Posteriorly, signals will be sent to the feedback application to move the subject 
through the maze accordingly. Additionally, thanks to pilot studies it was found 
that speaker location will be translated to movement in a better way if the last 
three classifications are taken through the control strategy instead of the five de- 
scribed in this experiment. The reason for using five was to make the EEC data and 
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performance more comparable between the experimental conditions (Five in the con- 
tinuous condition and five in the discrete condition). An additional step is necessary 
to achieve stability however, classifiers output will be passed through a sigmoid func- 
tion thusly making it less responsive to values closer to zero; this is done to ensure 
a certain amount of certainty in the classification before performing any movement. 
The resulting values will be used as input to a virtual joystick that will perform 
the desired movement in the maze. Next the BCI procedure just described will be 
framed inside the BCI paradigm, BBCI toolbox as well as within the pecuharities of 
the software used 

3.6 Experimental Control Design and Other Tools 

The AMUSE paradigm, the paradigm chosen for the present study, is based on an ap- 
plication of pattern classification and machine learning techniques. Additionally the 
physical design made easier for participants to associate sound direction to feedback. 
To the BCI user, feedback was visually presented on a screen showing the application 
called: MazeWalker. Posterior movement generated by the BCI was translated to 
video via UDP - OSC signals originating in MATLAB. Subsequently, the programs 
GlovePie and PPJoy performed the bridge between the control strategy scores and a 
joystick-like motion shown in the program. In short, both programs were responsible 
for conveying and transforming the data that resulted in real time screen movement. 
MazeWalker is part of MazeSuite^ which is a free software application designed to 
create 3D environments. It was developed by the team led by Hazan Hayaz and com- 
piled in C-|— 1-. This tool permits the developer create 3D environments for later in 
navigational experiments. The program was originally designed for spatial navigation 
studies, namely, navigational research using infrared imaging (4). For the current 
experiment the Brain Computer interface was designed and ran in MATLAB. To 
^ . . . mazesuite . com 
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interface the BCI software to MazeWalker, additional software was needed^. Two 
programs were used to make the bridge: GlovePie"'' and PPJoy''. In principle, PPjoy 
is designed to be a virtual joystick taking inputs from a variety of external sources 
other than joystick. GlovePie works in the same way only that this one in particular, 
allows OSD signals to be received as input. The inclusion of GlovePie was to ensure 
that real Joystick performance used as a benchmark be comparable in magnitude to 
any signal coming from the BCI classifier. For more detail about the Open Sound 
Control specification or the concrete implementation in MATLAB see the OSC doc- 
umentation. Three mazes were used for experimental purposes as well as two 
mazes for familiarization. The purpose of the use of the three mazes was to increase 
in difficulty as mazes or levels went on https : //github . com/wacax/MasterThesis. 

3.6.1 BCI Spatial Control Strategy 

The next steps differed significantly from previous experiments that made use of 
the AMUSE paradigm such that the spatial control strategy made use of the values 
gotten from the classifications performed by the BCI, which, in turn, were stored in 
the buffer in a five times five window each corresponding to one of the speakers / 
categories. Five scores at a certain time were stored for navigational purposes, both 
the continuous condition and the discrete condition maintained the same window 
width with the only difference being the weights. They were changed for the contin- 
uous condition. In the continuous condition older classifications were assigned lower 
weight to make more meaningful the most recent classification without excluding 

^ . . . This additional step necessary to create the bridge might no longer be needed in future 
versions of Mazewalker because it will include Net support according to its creator 

^ . . . sites . google . com/site/car Ikenner/glovepie 

® . . . https : //github . com/wacax/PPJoy PPJoy sofwarcs development was abandoned by its cre- 
ator, a copy of the source code can be found this on thesis github repository 

. . . For further references about Opensound Control refer to: http : / / opensoundcontrol . org/ 
intr oduct i on- o s c 
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totally previous data. 

The theoretical principle directing the BCl spatial control strategy, the fifth step 
on the online BCI pipeline is that raw classifications coming from the BCI fiow in 
a seamless stream of data have intrinsic information indicating the location of the 
speaker that the participant paid attention to. In order to translate this multi- 
class information into a continuous joystick-like stream of data we decided, after 
doing some testing, take the classifications from the three latter iterations. However, 
for comparison purposes (with the discrete condition principally), the window was 
widened to five. This process yielded a matrix of dimensions: 5 by 5 containing the 
classification for each speaker during the last five rounds. In order to get a continuous 
composed vector in two dimensions containing a value in the axis Y, meaning thrust 
and in the axis X, meaning heading the following method was chosen: Firstly a 
multiplication stored in the buffer or buffered classifications times some predefined 
values took place so the continuous condition can deal with the streaming data. The 
three most recent values buffered werent penalized whereas the two remaining were 
multiplied by 0.2 and 0.1 respectively to assure that as little information as possible 
from old classification scores was conveyed to the final calculation. 



weightsVector 



0.1 0.2 1 1 1 

0.1 0.2 1 1 1 

0.1 0.2 1 1 1 

0.1 0.2 1 1 1 

0.1 0.2 1 1 1 



(3.1) 



times 

bufferedData is a 5 times 5 matrix containing the results from the classifier or: 
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bufferedData 



Xii Xi2 Xi3 Xi4 Xi5 

X21 X22 X23 X24 X25 

X32 2:33 X34 X35 



(3.2) 



0:41 X42 2:43 2:44 0:45 



2^51 2:52 0:53 2:54 2:55 



In the discrete condition there was no initial penahzation because all five rounds 
provided equally valuable data since the participant was asked to pay attention to the 
same speaker in order to move to the desired direction as opposed to the continuous 
condition where intentions can change on the fiy. 

Secondly, the product of this penalization was subjected to a filtering through a 
sigmoid-shaped function. The Sigmoid membership function ^ is calculated on the 
values of the product corresponding to the product of weights Vector times buffered- 
Data, this step is necessary to ensure that small values that don't reach a certain 
threshold cannot be transformed into movement, thus getting rid of non-relevant 
data that was probably misclassified or appeared in the stream due to other circum- 
stances. 

Values "a" and "c" are tunable data to ensure that the output hmits of the 
sigmoid function fall within desired values 



Where the value x belongs to every single element of the weightsVector times 
bufferedData matrix. 

Moreover the additional step that had to be performed, given that values might 

jump from positive to negative and vice versa when the result of the aforementioned 

multiplication approximates zero but is not exactly zero. Then a sigmoidal-shaped 

^ . . . The sigmoid membership function was picked because it can be tuned to its lower value be 
around zero and the maximum value be 1, the concrete implementation was inspired by the formula 
found here: http : //www . mathworks . de/de/help/f uzzy/sigmf . html 



f{x,a,c) 



1 



(3.3) 



1 -I- Q-a{x-c) 
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function was included to smoother the movement when the classification was not 
clear nor the values were large enough, making the output of the function a value 
close to zero. However, when there was indeed a clear classification, the function will 
output a non-lessened value and even if its input value was high compared to other 
subjects; it also will be rounded toward one. As a consequence, filtering through 
a sigmoid function softened movements making them clearer and less sensitive to 
misclassifications, outliers or environmental fiukes when classification was not clear 
enough. Additionally, it capped the velocity to a maximum value when the classi- 
fication values had gone too high. This also helps achieve motionlessness when the 
participant was not paying attention to any direction. The whole implementation in 
MATLAB can be found linked in the experiment's repository on github. 

Thirdly, five sets of weights conforming to the directions indicating where to travel 
were selected based on performances of pilot experiments. Each of the directions' 
weights will correspond to a particular speaker, which, in turn, will have to be 
converted into a discrete value in X and Y axes respectively. The speaker will have 
two values: one corresponding to X and one corresponding to Y. The following 
nomenclature matches the figure 3.1 with the spatial distribution of the speakers 
in space. These directional weights will be fixed so that an iterative remapping to 
a XY space, 2 dimensional axes, can be performed. It can also be thought of a 
dimensionality reduction thanks to a matrix multiplication between values from the 
buffer and weights corresponding to XY axes and ignoring the non-relevant values. 
The axis Y would match up to forward movement whereas positive axis X would 
correspond to a right turn and negative axis Y would correspond to a left turn. 

Front - Forward, Speaker 1 

Value in X = 0; value in Y = 1.25 
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Right Turn plus Forward Movement, Speaker 2 

Value in X = 0.75; value in Y =1 

Turn Right, Speaker 3 

Value in X = 1.25; value in Y = 0 

Turn Left, Speaker 4 

Value in X = -1.25; value in Y = 0 

Left Turn plus Forward Movement, Speaker 5 

Value in X = 0.75; value in Y = -1 

Directional values can therefore be expressed as a matrix of dimensions 5x2. Let's 
call it from now on: directional matrix. 



DirectionMatrix 



0 


1.25 


0.75 


1 


1.25 


0 


-1.25 


0 


0.75 


-1 



(3.4) 



The directional matrix's values shown here can be changed according to the 
particular needs of the setup; though, the positive and negative values ought to 
remain unchanged for the BCI to move toward the desired directions. 

Next the remapping is done with a matrix multiplication: 



f(x,a,c)u f(x,a,c)i2 f(x,a,c)u f(x,a,c)u f(x,a,c)i-, 

f{x,a,c)2i f{x,a,c)22 f{x,a,c)23 f{x,a,c)24 f{x,a,c)25 

SigbData= /(x, a, 0)31 /(x, 0,0)32 f{x,a,c)33 /(x, a, 0)34 /(x, a, 0)35 

f{x,a,c)ii f{x,a,c)i2 /(a;, a, 0)43 /(x, a, 0)44 /(x, a, 0)45 

/(x,a,c)5i /(x, a, 0)52 /(x, a, 0)53 /(x, a, 0)54 /(x, a, 0)55 



(3.5) 



times 
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DirectionMatrix 



0 
0.75 

1.25 

-1.25 

0.75 



1.25 

1 
0 
0 

-1 



(3.6) 



The result of this matrix multiplication will be a 5x2 matrix, each row with a value 
relating to X on the first column and a value relating to Y on the second column. 
Posteriorly, the arithmetic mean of the values is calculated for the 5 iterations and 
the result will be a vector of length two with one value for X and another for the Y 
axis. This result will be the XY score at a particular time. 

PacketValue(t) = [Value in X Value in Y] 

Additionally, and this assertion is probably the most important differentiation 
between discrete and continuous conditions is that, in the discrete condition, instead 
of yielding a numeric value with any value from zero to one, it will yield a value of 
either one or zero. So if the discrete condition is selected after the mean calculation 
aforementioned, only the sign is taken into consideration because it only will be able 
to output of either positive or negative one or zero. 

Packet ValueDiscrete(t) = [binary Value in X binary Value in Y] 

3.7 Frechet Distance 

Commonly used for other applications such as writing recognition, the Frechet Dis- 
tance makes a good metric for the present trajectory anahsis. The difficulty of cal- 
culating two curves with the same length or with the same amount of points reside 
in the distance metric used; different methods have been proposed for that matter, 
when comes to curves without the same length; however, a different approach must 
be taken because other distance metrics assume two lines with the same amount of 
points in a given coordinate space. For this issue there are two alternatives at hand. 
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in video, audio or any other type of combined media the most common method is 
the dynamic time wrapping. For instance: three songs, three sets of waves, the first 
one recorded in a concert, a second recorded in a studio and a final one at a different 
studio will have a smaller distance between their waves when compared to others 
such as another song or the recording of a train passing by. Following the same 
train of thought we realize that measuring two curves is possible by comparing their 
dissimilar lengths in a systematic way. Given that the trajectories traced by the par- 
ticipants will only contain data in 3D, there is no need of going as far as calculating 
their distance using dynamic time wrapping. We ought to, instead, make use of a 
simpler metric the Frchet Distance because we are not interested in the analysis of 
the subtleties of sound and/or melody that such a procedure can entail. 

This metric, which can map many points in time to a single point in the optimal 
path, it can be better understood through the following analogy: when a dog is 
stroUing with its owner, both are hnked by the dog's lash. The owner can be standing 
at a particular spot while the dog can wander around to many different places at 
various moments. The different places the dog can wander will be mapped to a single 
point on the trajectory of the owner, namely, the closest one (imagine when the dog 
owner is standing still) . In the experimental case this will be mapped to the optimal 
path, thus giving us the following: 

The reparametrization between the boundaries of 0 and 1 is needed for this metric 
so all values at time = t will be within the range of 0 and 1; d being the distance 
function, in this case we will keep it simple so we will maintain the Euclidian distance. 

F{A,B) = inU,^maxt{d{A{a{t)),B{P{t)))} (3.7) 
In a thorough yet simple explanation: 

The Frchet Similarity index is equal to the Infimum (or lowest value) of the highest 
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number at time = t belonging to the interval between zero and one (reparametriza- 

tion) defined as: 

t e [0, 1] (3.8) 

, thus d or the Euclidian Distance of the point a of trajectory A at time t or A(a(t)) 
and the point b of trajectory B at time t or B(b(t)) will be repeated for all the points 
on the trajectories^. 

A detailed example is presented in fig. 3.4: 

The Frechet distance in this example is the mean of all the distances between 
points, the Prchet distance can be seen as the yellow fine in the lower left corner 
being compared with the size of the original curves. This method has been discussed 
in Aronov A. et.al Frechet Distance for Curves, Revisited. 



^ . . . The concrete implementation was taken from http : //www .mathworks . com/matlabcentral/ 
f ileexchaiige/31922-discrete-frechet-distance 
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Discrete Frechet Distance of curves P and Q: 2.6067 




Figure 3.4: Discrete Frechet Distance calculated between two trajectories. The 
pink line representing the overall Frechet Distance value between curves Q and P 
represents the mean of the distances between all the points pertaining to both tra- 
jectories,. (2.6067) here represented in light gray. 
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4 



Analysis and Results 



4.1 Experimental Contidions 

BCIs performance was calculated across carefully chosen conditions that compared 
a baseline or a benchmark with random results and with experimental conditions' 
results: 

4.1.1 Random Condition 

The random condition is the result of 120 runs of the experiment without a par- 
ticipant. Although in this condition synthetic data was sent from the BCI recorded 
from a participant during the pilots. The data fed was produced by picking randomly 
one sample of certain category ( or a label corresponding to a given speaker) and 
repeating it for two seconds in order for the data to accumulate in the buffer and be 
able to move, given that the use of a shorter sequence would have made impossible 
to move as stored samples would have been too small and even close to zero due to 
the remapping by the logistic-shaped function. See p. 20. 



27 



4- 1-2 Joystick Condition / Benchmark Condition 

A joystick was used as benchmark given its familiarity to many participants and easi- 
ness of use. Two repetitions of the same level were used to get a better approximation 
to average performance of the level by the current participant. 

4.1.3 Discrete Condition 

The Discrete Condition was recorded when participants used the auditory BCI with 
five iterations for each speaker and then the BCI system decided a direction to move 
to. It can only be a binary value of zero (no movement in an axis) or positive or 
negative one (full movement in an axis). 

4.1.4 Discrete Condition 

The Continuous Condition was devised in such a way so movement can flow seam- 
lessly opposed as the Discrete Condition in where there has to be a stop for the 
participant to listen to the stimuli and then decide and classify the direction to 
where he intends to move. 

4.1.5 Discrete Trimmed Condition 

This synthetic condition was created offline with additional processing. It is similar 
to the Discrete Condition in the sense that its trajectory remains the intact but with 
the only difference is that all time intervals in between movements were deleted. 

4.2 Curve Similarity Analysis 

The mechanics of this analysis is pretty simple; the Prechet distance calculation 
between two curves yields a value of similarity. Consequently in this metric, a lower 
numeric value is to be thought as an indicator of a higher similarity between two 
curves, the more similar two curves the lowest the Prechet distance. The objective 



28 



of the first part of the analysis is to compare both BCI conditions performances to 
the benchmark in this case using a commonly used input mctliod, a joystick. The 
distances can be deemed as vahd because the distances between the curves in the 
joystick condition and the random condition is the highest, as was expected; the 
experimental conditions are supposed to fall somewhere below random performance 
when compared to the benchmark. For this part, the Frechet distance was used for 
its ability to give a value to two curves of different lengths and more importantly, 
between two curves that can move freely and even come back to the initial point as 
it happens normally in movement trajectories. 

4-2.1 Condition Comparisons 

Random Benchmark: Firstly, in order to check the validity of the benchmark condi- 
tion, we used all the 120 trajectories and calculated the Frechet Distance, the simi- 
larity between random distances and benchmark distances, thus, having the highest 
scores when these two conditions were calculated. There were occasionally lower val- 
ues represented on the plot as outliers, which, of course, represent trajectories that 
thanks to luck got further than regular random trajectories and even some reached 
the end of certain rooms (also named as regions). These data points were out of 
three standard deviations. It is worth noting, however, that there are no higher out- 
liers due to the fact that the time was limited rendering impossible to create outliers 
with higher Frechet distances because the time was capped and therefore no longer 
trajectories could be produced. 

Continuous Condition Benchmark: The Frechet distance, i.e. the similarity 
between two curves, in this case continuous trajectory was compared to its respective 
joystick trajectory for that particular house (ex: continuous condition trajectory in 
house one compared to joystick in house one for that particular subject and so on) 

Discrete Condition Benchmark: The Frechet distance, in this case discrete tra- 
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Table 4.1: Statistics corresponding to Prechet distances described by condition 



Metric 


Disc-Bench 


Cont-Bench 


DiscTrimm-Bench 


Rand-Bench 


mean 


28.66 


25.54 


25.83 


38.55 


minimum 


6.62 


8.09 


2.71 


8.73 


maximum 


36.86 


47.60 


35.21 


50.48 



jectory was compared to its respective joystick trajectory for that particular house 
(ex: discrete condition trajectory in house one compared to joystick in house one for 
that particular subject and so on). 

Discrete Trimmed Condition Benchmark: The synthetic trajectories previously 
created based on the discrete trajectories for each subject was compared to the 
benchmark trajectories in order to check whether the trajectory without breaks was 
actually superior in accuracy than the continuous condition. Also this comparison 
can also be thought of as a potential performance for the continuous condition if a 
superior and more sophisticated control algorithm were developed. 

4.3 Results 

For the results of this comparisons of similarity between curves, all the results for 
each subject in a certain condition was grouped by taking the Frechet Distance 
of a condition and the Benchmark performance for that particular participant (ex. 
Continuous Condition in House one vs. Benchmark Condition in House one for that 
particular participant). Additionally all three results for each participant in a single 
condition (ex. Continuous in the houses one, two and three) were grouped with the 
results in the same condition for all seven participants giving us a total of 21 Frechet 
Distances within each condition. 

The bars in fig. 4.1 represent the similarity between the experimental conditions 
and the benchmark. There is also present a bar representing the similarity between 
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Ftechet Distances - Full Trajectories 



45 - 
40 - 

35 - 
5 30 - 
3 25 - 
E 20 - 



Figure 4.1: Frechet Distances between experimental! conditions accross all sub- 
jects. BenchDisc: Benchmark-Discrete comparison, BenchCont: Benchmark- 
Continuous comparison, BenchDiscTrim: Benchmark-Discrete Trimmed comparison, 
BenchRand: Benchmark- Random Trajectories comparison 

the random condition with its 120 trajectories and the benchmark trajectories. In 
the latter case there wouldnt be 21 data points but instead 420 each of which cor- 
responding to one random trajectory compared to each participants trajectory, (i.e. 
seven participants times six joystick trajectories times 120 random trajectories). 

The explanation for the similar scores in fig 4.1 for the continuous and the discrete 
condition in the boxplot is that: even though the continuous condition has an effective 
time of free movement of ten minutes, the actual limit of time in the experiment. 
Tthis longer duration accounts for the results because it allows the participant to 
travel much further compared to the discrete condition that has a time of 1.5 minutes 
of effective movement, having the rest only to stand still and wait for the BCI to 
classify the brain waves. 

For the same reason the Discrete Trimmed Condition was created, notwithstand- 
ing, in order for the distances traveled to be more visible and statistically evident, 
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Table 4.2: Statistics corresponding to Prechet distances in Trimmed Trajectories 
described by condition 



Metric 


Disc-Bench 


Cont-Bench 


DiscTrimm-Bench 


Rand- Bench 


mean 


29.8 


29.34 


20.83 


31.48 


minimum 


8.68 


11.89 


3.28 


0.4 


maximum 


32.92 


43.40 


40.21 


50.48 



it is also needed to take the same amount of time available in the discrete trimmed 
condition and shorten the other conditions to the same times (or trajectory lengths). 
In other words if we were to compare the discrete trimmed with the continuous con- 
dition, the continuous condition will still have an advantage in Frechet scores and it 
will be more similar to the joystick condition because it had more time to roam freely 
to be more like the joystick trajectory. The Trimmed Discrete Condition couldn't be 
more similar to that line when both complete lines are considered because in spite 
of its higher accuracy it cannot be further elongated. 

So the Trimmed Experimental Conditions were created by calculating the fastest 
Discrete trajectory and tacking the same amount of data from the other trajectories 
until that time. Put it differently, the two experimental conditions: Discrete and 
Continuous, were cut at the time in seconds as of the fastest Trimmed Discrete Con- 
dition in order to include all the Trimmed Discrete Conditions, thereby having more 
comparable trajectories in length, excluding the extra time the other trajectories had 
to outperform in accuracy. 

So the same calculations were performed using the same conditions but with 
the only difference that just first segment of the trajectories was used. The results 
differed in the next grouping as it can be seen in fig. 4.2 

The bars corresponding to the experimental conditions as well as the bar corre- 
sponding to the random vs. benchmark comparison remained at the same height, 
meaning that the trajectories are not significantly different among the three if only 
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Frechet Distances - Trimmed Trajectories 
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Figure 4.2: Frechet Distances between experimental! conditions accross all sub- 
jects at the time it took to the fastest Discrete Trimmed Trajectory for that partic- 
ular subject to complete. BenchDisc: Benchmark-Discrete comparison, BenchCont: 
Benchmark- Continuous comparison, BenchDiscTrim: Benchmark-Discrete Trimmed 
comparison, BenchRand: Benchmark-Random Trajectories comparison 

the time needed for the Trimmed Discrete Condition is used. The Trimmed Discrete 
vs. Benchmark comparison, however, stood out to the other comparisons having a 
lower distribution in comparison to the benchmark, meaning that it was much more 
similar to the joystick (benchmark) condition when only the first lapse was used in 
the calculations as when the full time trajectories were used in the calculations. 

The explanation here is rather simple, the experimental trajectories other than 
the Trimmed Discrete, take longer to develop a similarity to the benchmark whereas 
the Trimmed Discrete resembles more the benchmark and takes less time to be more 
similar to the benchmark also, this is certainly a clear indication of higher accuracy 
in the Discrete condition in general, where, despite its lower speed not because the 
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speed was capped but due to the pauses in between movements had more precision in 
movements often comparable to the benchmark. This result can be also understood 
in terms of potential development of this technology, it tells us that with a more 
sophisticated control algorithm higher performance is possible and as we will see 
later on even comparable to the joystick performance in both accuracy and speed. 

4-3.1 Preciseness of Full Trajectories and Trimmed Trajectories 

An additional way to visualize the previous effect is the following figure (4.3); here 
the graphics from an upper perspective have an underlying heat map indicating 
population density. Each point here represents a location where each participant 
was located during the whole experiment, in each subplot, a different condition is 
shown. For the random condition 10 trajectories were randomly selected in order to 
plot the lines more clearly. 

The continuous condition resembles more the trajectories of the benchmark, how- 
ever, if a cut is made at the minimum time needed for the fastest Trimmed Discrete 
Trajectory to finish the image changes dramatically. See fig 4.4. 

In figure 4.4, where, minimal values for each participant are chosen, the trajec- 
tories of the Continuous and Random are alike because the continuous Condition, 
due to its lack of accuracy it does not have the time to advance in the mace and 
had a hard time bumping onto walls mainly whereas the more accurate condition, 
the Discrete one, is able to do a better job in a shorter interval, with and without 
trimming the moments when the movement ceased. 

4-3.2 Time and Distance Analysis 

The two metrics were extracted indirectly from the log files using the MazeAnalyzer 
software, part of MazeSuite, which also takes into account natively the experimental 
maze with its objects within and the trajectories, so no additional preprocessing was 
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Discrete 




Figure 4.3: Upper perspective of the full trajectories, hottest colors denote higher 
data density around that pixel 

needed. 

The distance is once again measured in Maze units and the time in seconds, 
the velocity is relative to both these measures so it is calculated in MazeUnits over 
seconds. For this analysis the maze was separated into six regions corresponding 
to the following figures for each house. The reason for that was: path length and 
time taken from the complete trajectories will be heavily influenced by the time cap, 
meaning that time will tend to lean toward 600 seconds for the unfinished mazes as 
well as a certain length, especially, in the discrete condition where in addition to a 
time cap, there is a moment of immobility, producing similar lengths in unfinished 
mazes. 

As a consequence, the mazes were split into regions to take the time and path 
length in them, i.e. the time spent and the distance traveled before moving to the 
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Continuous Trimmed 




Figure 4.4: Upper perspective of trimmed trajectories, hottest colors denote higher 
data density around that pixel 

next region, giving us a better approximation for an uncapped time limit as well as 
distances that vary according to accuracy and intention of movement. 

Given the reasons outlined in the previous paragraph is predictable that the 
most accurate and precise conditions will have lower time and shorter distances in 
region one. The results agree with this assumption as is shown in figure 4.6. Here 
the performance of all subjects in region one is presented with the times and path 
lengths as metrics. Region one has been selected because its dimensions and layout 
is exactly the same in all the three houses and it is relatively simple to navigate 
through. 

It is important, however, not to assume the applicability of the same reasoning 
for all the regions because they vary in different ways such as in objects found inside 
and even sometimes dimensions. It is worth noting though, that similar results were 
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Figure 4.5: Mazes' layout, the three mazes had exactly the same dimensions 
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Figure 4.6: Length of the path in maze units in region 1 for all subjects and all 
houses separated by condition 
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Time Elapsed in Seconds 
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Figure 4.7: Time in seconds spent in region 1 for all subjects and all houses 
separated by condition 

obtained when the second and third regions data were collected and calculated, not 
in the sense of same times and path lengths but in the sense of same distribution 
and performance compared to other conditions 

Table 4.3: Statistics corresponding to Path length and Time in the first region of the 
maces. 



Metric 


Path Length Cont 


Time Cont 


Path Length Disc 


Time Disc 


mean 


402.23 


38.11 


299.67 


36.68 


minimum 


28.27 


10.27 


41.01 


9.88 


maximum 


599.97 


94.9 


599.91 


75.66 


skewdness 


-0.58 


0.74 


0.14 


0.24 
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4-3.3 Projection onto the optimal Path 

In addition to the previous analysis, a quantitative way of measuring performance 
in percentage is to project the performance of the experimental conditions onto the 
best possible path and visualize in this way how much it traveled compared to the 
fastest conditions (Figure 4.8). The procedure for getting the optimal path is rather 
simple, take the lowest time that a benchmark condition took to finish the maze and 
sample the same amount of data from the other benchmarks from other subjects, and 
extract the mean of the points. Later, take the experimental conditions trajectories 
at a certain time and look for the closest point in the optimal trajectory by calculating 
the Euclidian distance between the data points of the optimal condition and the point 
of the experimental condition at a given time, the minimum distance will be the 
closest to the optimal trajectory, thus having the nearest point to the experimental 
condition at a given time projected onto the optimal trajectory. 

Aindex = infa{d{A{a{t)), P{t))} (4.1) 

The mathematical description of the process abovementioned is shown here where 
A is the optimal trajectory and the point labeled as "beta" represents two points 
in X and Y respectively that we want to check for the smallest value or infimum 
across the curve. Moreover, in each condition multiple data points will be present 
from different participants so the lower and higher bound as well as the standard 
deviation of these projections will be calculated. 

4.3.4 ERP Analysis 

A standard ERP analysis over the training BCI data was conducted. The objective 
of the analysis was to present evidence that the P300 waves presented a similar 
behavior as the previous studies that such data was used for classification purposes 
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Conditions at 37 seconds, Dispersion 
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Conditions at 37 seconds, Percentage of maze completed 
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Figure 4.8: a) Dispersion of all participants in all houses projected onto the closest 
point in the optimal trajectory, b) Means across all subjects, c) Percentage of 
Maze completed at t:37s. The reason this time was chosen is at 37 seconds exactly 
is the value when the quickest trimmed trajectory would have finished the maze 
and coincidentally the fastest joystick trajectory also ended at 37s. The mean and 
standard deviation values for the boxplot are the following: stdDisc: 3%, meanDisc: 
8%, stdCont: 6%, meanCont: 6%, stdDiscTrim: 34%, meanDiscTrim: 50%, stdRand: 
4%, meanRand, 3% 
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in a speller. Results of the participants performance were looked for correlation but 

due to the number of participants being lower than ten a correlation is difficult to 
find. 

Time Filter 

The participants were exposed to stimuli in order to elicit P300 waves, a high pass 
filter was used to reduce drifts, moreover the electrodes in the middle were the 
ones selected to plot the main characteristics of the classification data. The artifact 
rejection was done based on variance criterion meaning that the ones found over three 
standard deviations were removed from the data. Additionally the filtered data was 
segmented; each segment or epoch was concatenated one on top of the other, and 
averaged. 

Targets and non-Targets 

The distinction between targets and non- Targets after filtering was looked for in the 
lapse between 200 Ms. and 400 Ms; the lapse corresponding to the P300 varied across 
subjects as can be evinced in figure 4.9. The mean Prechet scores for the continuous 
condition is presented next to the figures showing the P300s near the electrode Cz. 

ROC scores were used to separate performance in subjects in calibration data 
shown next. The advantage ROC scores has is that is useful to quantify the [sep- 
arateness] of one dimensional distributions (binary scores) a useful metric to take 
into consideration in a five class classification. This score has also been taken in 
consideration by Schreuder(17). 

Though an important detail not shown, is the persistence of eye movements in 
the online data, in a navigational study eye movements will be a major issue as 
participants cannot help but move their eyes because they will always be checking 
position on the screen and it was a cognitive load that was tried to be lessened by 
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Cz 




Figure 4.9: ERP average across epochs by nbb; P300 Classifier accuracy: 68.6%, 
average Frechet distances Cont: 25.54. Typical ERP plot among subjets. 

making them rehearse in the actual maze so that the can recall the position of the 
objects and rooms without having to move much their eyes, but there is no clear 
option in where participants wont move their eyes in online data. 

An additional detail worth including is that: although the number of participants 
was not big enough to find correlations in, a trend among participants was found 
showing that they found significantly more difficult the continuous condition followed 
by the discrete one and the joystick condition. This information was obtained with 
the use of a follow-up questionnaire. 
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Figure 4.10; Event Related Potentials grand average, a) Raster Plot of averaged 
epochs showing increased activity around 300ms. b) Activity of electrodes around 
the scalp, for a higher resolution of that image, see on Annex 1. C) ROC score for 
Target-Non- Target peaking over 0.04 on the location of the electrode CPz at 300ms. 
Additional scores with scalp maps over time, showing the presence of eye- movement 
certainly due to navigational necessities 
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Conclusions 



Here is presented a method for non-invasive navigational for a brain computer in- 
terface design that uses ERP classification to navigate through 3D environments. 
The results presented in both psychometric and electrophysiological analysis of the 
implementation of a ERP based BCI point at the potential future implementations 
of such technology. The results demonstrate the feasibihty of the use of P300 as well 
as ERPs being classified and later buffered in memory to generate output in virtual 
environments. Optimization based on the output from the classifiers is likely to be 
improved by the use of more sophisticated algorithms. Participants with accuracy 
circling 80% in the classification/calibration phase outperformed other participants 
with lower scores suggesting the impact of PSOOs in the overall control of the designed 
BCI. 

As classification accuracy increased, the Frechct distances decreased; so did region 
times and path length. The break between the targets and non-targets in the epoch 
grand mean is present indicating that the P300 activation was the responsible for 
the activity and it corresponded to activity in the brain similar to the ones reported 
by Schreuder (2010). 
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Overall, we can state that a P300- ERP based navigational BCI is possible to 
create, for mainly two reasons: P300 based BCI has proven to be effective among 
different tasks mainly communicational, take AMUSE as an example; Taking these 
classifications into the spatial arena is a step forward into developing new BCI tech- 
nologies. As it has been shown, the main pebble on the road is the creation of 
meaningful control mechanisms to translate these classifications into movement. It 
has been long known in Neuroscience and Artificial Intelligence that movement and 
vision, two human capabilities that seem so clearly easy for humans or animals to 
perform are in practice two of the most challenging engineering problems that re- 
searchers can face. Consequently, BCI technologies that have to deal with similar 
problems in the future might take that into account when developing movement that: 
a) cognitive load in the subject shouldnt be heavily burdened so that a BCI appli- 
cation is possible and b) control algorithms that take P300 signals or whole ERP 
data should have an embedded degree of intelligence, possibly smart systems that 
can change output by changing the navigational algorithm on the fiy. 

A closer analysis of the P300 navigation BCI performance showed mainly that 
P300 for continuous stream of data is possible although it may require tuning to 
make it more precise. Additionally a discrete P300 condition that classified step by 
step has proven to be more accurate and when it is trimmed as its accuracy and its 
speed in top performers is rather comparable to the benchmark condition, suggesting 
that control based in P300 is possible and it is hkely to have better performance 
boundaries. 
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Appendix A 



Various Addendums 
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Figure A. 2: ERP average across epochs by nba; P300 Classifier accuracy: 72.6%, 
average Frechet distances Cont: 28.51 




Figure A. 3: ERP average across epochs by nbc; P300 Classifier accuracy: 58.46%, 
average Frechet distances Cont: 35.91 



47 



Cz 




} ii * @ # [ ' 

Figure A. 4: ERP average across epochs by nbd; P300 Classifier accuracy: 66.76%, 
average Frechet distances Cont: 27.17 




Figure A. 5: ERP average across epochs by nbe; P300 Classifier accuracy: 54.86%, 
average Frechet distances Cont: 33.9 
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Figure A. 8: Layout of the second experimental Maze, upper perspective 




Figure A. 9: Layout of the third experimental Maze, upper perspective 
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